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_ * • SUMMARY . ■ ' • ■ . 

The BASIC Instructional Program (HP) was developed to investigate 
tutorial modes- of interaction in compute r-assisted instruction (CAI). 
BIP is an interactive problem^solving laboratory that offers tutorial • • 
assistance to .-students solving introductory programming problems in' the 

/ BASI C language . The' p^roblems- are presented in an individualized sequence. 

_bas*ed on a representation of the "^structure of the- curriculum and a model 

"of the student's state of Icnowledge. 

: After a brief review of tihe. rationale and origins of the BIP instruc- 
tional system, the design and implementation of EEP^s Curriculum Information 
Network (CIN) are described. The CIN stores the relationships among 
elements of the author-written course material. Each problem ("task") 
is linked,.:in the network to the programming skills required in its" 
solution. The" entire _ curriculJim is represented in terms of these skills, 
which are grouped into qubgoals representing the fundamental programming 
"techniques." The task selection strategy uses the CIN and the technique 
hierarchy to guide the .student through the curriculum. The student's 
history of success and failure on the skills involved in his previous 
?)roblems is used^ tO'^ i^Afy areas where more, work is required.' An . 
appropriate ."next task" is then searched for in the CIN. 

The environment for th^se instructional ^iecisions is the tutorial \ 
laboratoiy itself. The BIP system is. described briefly, ..with emphasis 
on recently developed features. The goal of the tutorial laboratory is 
informative interaction with the student, whic^ is provided by. an in- 

V.'. 

structional BASIC /interprejber, information on BASIC 'syntax cross- 



■ "referenced with the student manual, and debugging aids. The system, a Is. o 
"has access through .the CIN to features that the student may use to. help 
him complete his current task. These features include hints, easier* 

■ . ■ • -»/ ■ 

"subtasks," a stored solution that ' can. itself be 'executed , and an inter- 
active graphic representation .of the solution/ The nature of l;he student- 
BTP interaction is captured in an annotated student ^dialogue illuistrating 

8 typical Session. * , * . " ' . * . ' 

.'" 

FJ.xially^.jihe,.-re^Jilts of a . controlled experiment .are reported. BIB's 

task selection process was compared to a fixed 'linear path through the 
same curl:iculuin; though the total amount of learning appeared to be 
unaffected by the. treatment groups, the nature of the learning experience 

^provided by the two problem selection schemes was quite different in 
some interesting respects. .■ . ' . ' . 



THE COMPUTER AS A TUTORIAL LABORATORY:' - " ' 

. . / THE STANFORD- BIP PROJECT^ 

Ayron Barr, Marian Beard, and Richard C. Atkinson ^ : 

- ■ . . " * • - ■ . ' ■ 

- • ' • •' ■ * " ■ ■ ■ . • ■ . ■ 

I. Background ^ ' 

■ ^ Computers are now used in a; >wide variety- of applications in educa- 
tion and training, including information presentation and drill, • 



information retrieval, and simulation of complex systems. OSie^.e^'earch 
•reported here deals, with an additional application: the us^,of tlrjfe 
• computer as a problem solvi-ng laboratory.-; In the compute r-.based laborat 
environment, the student attempts to'-solve problems on-line with the 
, guidance of the instructional system. ' The system plays- the role of 
intera'ctlve tutor, gi\ang hint^s , correcting errors,, and' evaluating 
' progress.. ■ The full paver of '.the computer as calculator-. and simulator .• 
is available ..to the ^itudent, providing the motivational effects of -, 
learning hy vorklng \>ri real prhblems yith adequate bupervislon. at ■ the 
studexit.'s convenlfeuco" and at his ovm pace. The main focus of our work 
in the Complex Tnstractional Strategies research group at tlie. Institute 
for Mathematical ot-udie.'^ in- the Social Scieuces at -Stanford University ^ 
is y^e indivldu$.li^:dation ^of tne sequence of instruction presented in 
computer-assisted instruction (CAl). An effective CAI ■ program in a ' . 
given subject area can create a learning environment adaptively suited 
■ to. each student at his own level of -development. • ' . ■ \ 
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^The ^uthor6 thank Oliver Buckley, Richard. Kahler, Jay Lindsay, and 
William Sw^r tout for their contributions to BIP... 
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The computer-based tutor design/has been ^^rHyed at by severall • •' 
research .groups using^ different approaches to Artificial Intelligd<^e 
applications in CAT.-- Garbohell, Collins, -ahd others (Carbonell Sc Qollins, 
1973; Qollins, PaQsafiume, Gould, & Garbonell, I973) developed tAe GEO- 
SCHOLAR syptem to illustrate natural language inquiiy .of a« large, data 
base* However, the GEO-SCHOLAR' syGxem is really an elaborate tutor^^ith ^ 
sophisticated decision algorithms directing "mixed-initiative" dialogues:. 



the/ instructional^ system can ask questions- as w^il as answer them. Their 

.• ■ . « ■ ■■*■'»■',< 

rec^t'work explores tutx^rial schemes for instruction in more procedural 

■ ■ . • ' • , ■' * \ ^ ' 

su)Dject domains (e.g., an on-line editing system) vhetfe simply .asking and 

f •■.'.'■.■*.' - . ''•»■' 

answering questions is insuffieient (Grignetti, Gould^ riau'lsmann. Bell, 
Harris, 8c Pas safiiime, 1974). , 

■ Daniel^on and Nievergelt's work at the University of^llinois -PLATO 
system concentrates on -automated problem solving assistance -(Danielson, *^ 
19755 Dahielson 8c Nievergelt, 1975) • They use a top-dowi problem solution 
graph to , direct a tutorial di^alogue abcJut how to solve a programming ' 
probieiff; Although their system does not** build a model of the ^udent*^ 
frbm which- future instructional decisions -could be made, the problem • 

' ■ ■ ■ ' ■ ■ ' ■ ■ ' \ 

solution graph scheme lea'dp directly to a- useful representation of the - 
curriculum*. The student model could b^' updated as the student trave^;^ses^ 
the graph' in his attempt^to find, a solution ;^this is a procedure we will'/ 
incorporate in Cur Work on BIP's REP 'subsystem (d'escl'ibed in Section IV) 
in the coming year. " • • * ^ 

Perhaps the most impressive and "lyiowfe'dgeable" computer-tLased'' tutor 
yet devised is Brown's SOPHIE system (Brown, Burton, h Bell, *197l|) which^ 
grew out of research on modes of querying a simulations-based knowledge 



representation. . Although curriculum- guidance ' decisions "are minimize* 
(the-. system -Eeaehes only one skill, tro^bleshootin;|^ a complicated elec- 
tronic? circuit)/), SOPETE^s knowledge of t/roubleshoating strategy and . 
logical deduGtions from' known measurements, fosters "learnif% by imitatioii" 
in a natural and. exeiting' environment /the essence; of tutorial- style. 

^ -Research ^at^IMSSS" has approached the .compui:er tutor'model-- .sUcces- 
'sive refinement- of more^ traditional approaches to pAI iip logic -and compute 
■^ogrpiqiTrxng-. The'aogi^..and..jao.ffi.-.ad,^^^ courses no,^ running 



in fully tutorial mode were first 'conceived of as. ap|licaf ions, c^f auto- 
mated theorem proving .techniques for checking studerfts* proofs* (Qqidb.er&, 
1973). current work-in the advanced set .theory course that we offer at- . 
Stanf.ord involves infonaal, natural language/ student-machine di4logUes 

to' discuss^ develop ^nd refine complex mathematical pmofs (Sfl3^'|:i, Graves., 

* . . ^ - \ « - . ' • ' 1/ 

Blaine > 8c- Marinov, .W:5). , . . ' \ . " , " \ 

' 'Tn 1970 the jri^rtitute*'':? Complex Tnr. tract ional. Strategi^^ group .' 
* developed . a 'largri CAT cutTicaami fur a new course tu teach, ihe AID pro- 
grammtng. :Langu.ig..'.(lyob).ar twr'tntroductcay una^-rgraduate level. ♦This. 
course lu& beeti u-ieo. ■ i n culleg-^;-.; and junior. collc.ge..: as a succftssful. . 

" " " , a 

•■lntroductl.;ri t.- CumpJ-cer progratiiii.i.vig; ( l--xierid,_^ 1973; 'Heard, Lorton, Searle 
"I •Atkinson; 'I'^'S;- liiW-r, it is a lirieax:, ■''fraaie-orierited" CAI program 
and cannot provide' .individualized, ins function during' the- problem-solving 
ac'tivity itself. 'After working through lesson segments in such topics 
as syntax ana expres-^ions,. the stude.nt U assigned a problem to solve in 
•-aid/ .He must the^' leave the instiucLiunal program,, .call, up 'a separate 
. AILl interpretfer, pericrm the ^required progfairming tarflc^ and; return to' _ 
thi instiuctional' program -vd-th an answer. As he develops his^program • 



dlrectly*^itJf''3!\fD^.. sourbe of asslsta^nce is the mll^imally Infor- 

raative error roessage's provided by the interpreter, ' . 1 

furthermore., the AID cburse was found to be an Inadequate yehlcle ^ 
for more, precise investigations of in^ividualiaktlon of instruction 

•becaus'e'of the linear organiz'ationi of its curriculum. The course cbn- 

' ■ \ \ ' : ■ * : ■ * 

* .. ' , . ^ ■ ^ : y * . ^ 

slsts, of a- large 'set of^'rdered l^es^ons, re^iaews, and tests, and a . 

student's progress «jfrom- on^ segment to the next^was determined-^by his,^ 

score on "Che previous se'gmerit/. A high ycorq-'wduld lead to an "e^tra 

credit-* lesson ,pn the same' concepts, wmle'a low score would be followed 

« ' . . » . ^* • ' 

'bjr^a re-View Issson. it '.became cl^ar that this decision scheme, bas^dd . 

'* ' ■ . . , . . '• . • 

* ' , ' . .• * ' ■ ■ ' ' * . ' 

oVi total lesson scores, was reasonably eff^aiive in providing instruction 
add programming practice ,t;jbut since it d.ealt with rather large segments 

'oY the-qurrlculum, the iridivlyua-liz^ation oif the course of study was 
minimal, 'All students covered more or iess the same concepts in tl:i^ same 

. order,' with slight differences Jin the /amount of review. We wei^ inter^ 
ested in developing a system whose decisions would be based on a more 
specifically defined goal: the mastery of particular progranipilng , skills; . 

'rather than achievement of .a/ criteilon lesson score. For this reason, 

•9 V . 

we^ updertook development jof^e^jcgu^^ a new and different ;instruc- 

tidnal design, based in part, on earlier work bjr Paul Lorton (Lorton & ■ 
Slimick, \%9J. • ' \ ' y 

The bAsIQ Instructional Prdg^am (BIP) is a stand-alone, fully 
self-containe'cl course in BASIC \pyograimaing at the high school/college . 
level developed 'over the past two years with the assistance of over 300 • 
ijndergradaates who have taken the course at DeAnza .College, the University 
'of Sah**FranCliBCo, sand Stanford. EEP*s ma^or features are: 



^ A moriitored BASIC interpreter, written" in SAA (Van Lehn, 1973) 
' by the IMSSS staff, which allows.the instructional system 
* . maximal Imowledge abcut student errors. . 

- A curriculum" cohsistijig of approximately^lOO well-written, 
• . interesting programming problems at widely varying levels 
. of difficulty. • . " 

* "i A HINT, system,- - which gives both- graphic and textual aid in _ 
. ... problem solving.- ' ; : ' « 

' Inaividuali^ed task selection based on a .Curriculum information' 
' [ y^'-rsj^etwom,: which deBcribes^th^ problems; in--te^^^ 

mental skills* Prd'biems -are selected using a model or the 
. stu(^tnt'-s acquisition of the skills required by his .ear li-er 
programming-^problems. , .. . . 

Figure I'i? a ^ schematic representation , of the tutorial programming^ - 
laboratory environment supported by HEP, described, fully by JBarr, Beard, 

■ ant Atkinson (1975).! Section IV presents a brief description of the . 
system including some- hew featu^res not described „in the earlier report. 

V Section V :is an annotated dialogue Illustrating the system's features . 
and the stude-nt-EEP interaction. ' ' ■ 

. ■ The new work; reported Jie.l-e is primarily concernea with- BIP's 

' optimizea. problem selection .by means of - an: iixberrial representatidn . of 

• the\:'curriculuffi. Structure, the Curriculuni Infoiroation- Network (CIK) . The 
■' use- Of network-models to. describe . curriculum .structur/^^ is an important 

'•. deUlopment in tutbriai ■ GAI ; The .CIN enables the instructional program 

• t6 ."know" the subject matter it purports to teach/ and to meaningfully 
J_ 1^^^^^ tl^e linfes of his develeging skills,, 
• ■■ instekd of the curriculim elements (problems )^^ t^^ 

/section discusses the 'current state of curriculum design for CAI xiourse- 
' - -ware tte -sources and^ context; from which thfe CIW concept emerged , and ■ 
[bur implementation ocP HEP'' s CIK.: Section III ^discusses ' our use of the 
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network to" optimize task selection^ ana. de^fiEes" the--^orithr^^ 
rently in .use. Section VI describes an experiment comparing tTiese ^-^ 
algorithms with a fixed curriculiJim path for their effects , on student 
• perfomance, * , / ' 



11. Curriculmn Represeutati-Qja in CAT* ^ 

— lJUjmifilLj3f the current research in tutorial, CAI, generative CAI, * 

and mixed- initiative natural language. dialogues7"^t^ 

the "representation" of the' subject domain, which is*, also a fundamental 
concern of research, in cognitive psychology and artificial intelligence, 
Th6 goal is to provide a .representation of the subject matter that is « 

^sufficient for individualized tutoring and also has a 'realistic, and^ 
manageable computer implementation. A consideration of the different 
"representational poles" in vogue in CAI will give a perspective on the 
capabilities of the Curriculxim Information Network" representation, , - 
• • The most common style of CAI- courseware now being written consists 
of an automated presentation of a curriculum written by a .human author. 
The author, knowledgeable in the subject matter, has in mind a clear 
organization of the interrelations among the specific "facts" of . that 
subject,. an implicit understanding of the dependency of one .concept on 
another, and a plan "for the development of new^skills. His personal 
organizatipn of the discrete elements results in a strucrt:ured curriculum, 
consisting o:^* lessons or problems presented in a sequence he considers 
to be optimal in some sense for his model of his students. This structure 
is like that of a textbook, established in advance of interaction "with 
the student, but superior to a textbook in that the author builds 
branching decisions into the program, providing some degree of indiyid- 

'ualization. His subdivisions of the curriculum and the 'branching criteria 
he specifies constitute the. author's representation of the subject matter 
in^ this traditional CAT style, ' 



■ At the. opposite pole of explicit structural infomation are 

"generative" CAI programs, which do not use an author-written curriculum 
---at allT-^^3MM^-4ype-^ statements SLnd soli^tions 

-iDy-^4rie\djigJ£^om^ from a complete, internal representation of 
the facts in the subject domain,. usuali3r^ore~a:-in--a semantJ.jc.jietwoi^ 
Question-and-answer construction algorithms are used tp present the 
material in the data base to the. student. These algorithms also embody 
heuris^tjs fof what to teach when, depe.ndirig on some model of the 
siudeht^s" state of knowledge. All the "facts," dependencies, and logical 

. interrelati®ns that form the, author's knowledge of the subject must be 
embodied within the generative program. Tljus, Carbonell's Vell^knowh • 
SCHOLAR program ( Carbbne^, ; 19T0) "knows" the names, and populations of 
; the major cities of Brazil, : and is; clever enough to answer "What is the 

' largest city ; in Brazil?" without "knowing;^ that fact explicitly. 

^> The advantages for individuald^ation of generative CAI over fixed- 
branching courseware- are considerable: the generative program is 
specifically designed to provide instruction arid/or information in . , 
precisely' the' areas needed by the student. All decisions about what . 
material to present can be made dynamically, ' based on the student's 
progress gather than on- a predetemiined sequence of material. Ideally, 
the program has access to the same information^ that mak^s the human . 
author a subject ^lattei* expert, and this information/ can be >ade avail- 
able to the student much more flexibly than is possible in author- 
generated CAI. In particular, the model of the.' student's state of . 
knowledge is based on* Ihe structure of- the subject itself (e..g., the 

student has covered the. material on- rivers in Brazil) rather than on 

' . ■ • •» • • • . ' ■ 

*■ • * . 

. • . li . ' • ' ' / 

- ... \ . - - . . / . . . ■- 



thfe structure of the author*s curriculum design as reflected in his 
bi'anching specifications, which are typically triggered by correct/wrong . 
itesponse counters, . • 

.In a very simply s true turM quest i^^ 
based decision policy can adequately reflect student pragress. For 
iTistance7-.4JLjthe jrogram asks questions, about rivers in Brazil until the 
student answers two correctly in a row, then tliei^ iS" inde64>^-SQme^ 
fidehce about the student 's knowledge of that subject. However, this 
is exactly the type of course material that can-be progranj-generated*by 
current methods; unfortunately, both the simple quest ion-ai^d-answer and 
the program-generated approaches yield interactions that tend to be quite 
dry and unmotivating. The principal advantage of author-generated Courses 
is that they can be well written. The author's organization of the 
mateliaT*and: style^of va|tt±ng-~<3an,-he-P0Hei^ factors, - 

■ The Curriculum Information . Network 

In technical subjects, development, ^f skills requires the integra- 

*•'"■< > ' " ' ■ , ' ' ■ ■ . ' " 

tion of fa&ts, not just their memorization, and the organization of 

insti^ic'tional material is crucial for effective instruction in these 
areas." As^t'he durriculum becomes mo^e complex, involving the inter- 
relations of many facts, the author's ability to present it in a fomat 
that facilitates assimilation arid ir^tegration becomes more important. 
At the same time, hoVever, using counters to keep track of the student's 
progress through the curriculum provides a less adequate model of his 
acquisition of knowledge, ' ■ . ^ 

" The Curriculm Information Network is intended to provide the in- 
structional prograi^ with an explicit knowledge of the structure of an 



author-written curriculum. ' It contains the interrelations between the 

problems which the author would have used implicitly in determining his • 

"branching" schemes. It allows- meaningful modelling of the student *s ■ 

progress, along the lines of his developing^ skills, not just 'his history 

of right and wrong responses, without sacrificing the mbtlvatioriar r 

advantages of human organization of the curriculum material.* For example, 

in the HEP course, the' CIN consists of a complete description of each of 

100 well-written,, interesting programming problems in terns of the skills. 

developed, in solving the problems. Thus, the instructional program can 

monitor the student *s progress on these skills, and choose the next task 

with an appropriate group of new skills. An intemediate step , is intro- ^ 

duced between .re cordirg the student *s history and selecting his next 

problem: the network, .becomes \a model of the student*s state of knowledge, 

■ . V. ' - . ■ ■ 

•since it has an estimate of -his ability in the relevant skills, not just 

^his^ performance^ on the problems/he has completed. Branching decisions 
are basecj on this model Instead of being determined simply by the student* 
success/failure hls+L.-ry on the problems he has completed,^ ^ 
^' In this way, a problem can be presented for different purposes to. 

. students- with different histo'ries. The . flexibility of the curriculum 
is of course multiplied- as a result. More importantly, the individual . 
problems in the curriculum 'Can be more natural and meaningful; they do"^ 
not necessarily involve only one skill or technique. In frame-type 
currlculums this one-dimensionality of the problems has a constricting 
effect. In essence, the network "as implemerite(J in HEP is a method of 
describing a "real'' curriculum in terms of the- specific skills that can 
be Identified as a student *s problem areas. 



The next section describes ^BTP's implementation of the Curriculum 
infoimation Network and the algorithms which use it to .select problems 
for students in an individualized manner. 
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• III. Individualized Task Selection Using the'Netvork' 

Computer-assisted instruction has long promised to present' ah 
individualized sequence of curriculum material, but in many cases this 
has meant only that '*fast" students are allowed to* detour around blocks- 
of curriculum-, of \hat "slow" students are given sets of remedial .exer- 
cises. describing the curriculum in terms .of the skills on which the 
student should demonstrate competence, and by selecting tasks on Ahe^ 
bas^.s of individual achievement and/or difficulties,, we intend lo provide 
each student with programming tasks that are both challenging and in- 
structive. .Fu'rthemore, the structure used in ELP can be applied to 
many other subject areas (such as statistics, algebra, or reading) that 
involve identifiable skills and that require the student to apply those 
skills in different context's and combinations. . " 

We describe^the curriculum as a set of goals, ordered by a tree 
hierarchy. " In a subject that deals primarily with ?he fprmollation and 
—solution of problems^, as opposed tp the, absorption of factual infomation, 
a -curticulum goal is to be interpreted a^" the„ maste^ of a particular . 
problem-B Giving technique specific to the. subject matter. The desired 
end result, then, is the achievement of one or more top-level goals, 
each of which depends on one or more prerequisite goals. .Each goal will 
\bQ described in the program in terms of the acquisli:ion of a set of 
skills, and the problems, or curriculum elements-, are\ described in tems 
of the skills that 'must be applied .to solve them. A ski.ll may be . 
deve loped, in^are than one goal, and will moirt certainly -be used "in 
several problems. 



X,: :■ 



In BlP^ then, curriculiuuTi goals Involve the mastery of certain pro- 
gramming techniques. The techniques we have chosen, include: simple* 
output, using hand-made loops, using subroutines, etc.. We have cho.sen- 
for the purposes of our current research aOvery simple case of the. full- 
tree structureTTol^-^goals. " The techniques are- linked in a linear order, 
.each having but one "prerequisite," based on dependence and increasing 
program complexity. Other structures are attractive**, 'but our current 
research deals . primarily -with individualizing the sequence of presenta- 
tion of problems, once the cuKriculum structure has been specified in' 

the CIN. ... 

The techniques are interpreted as sets of skills, which are' very 
specific curriculum elements like "printing a literal string" or "using 
a'^counter vari.able In a 'loop." The skills' are not themselves hier- 
archically ordered. Appendix A is a list of the techniques and the 
skills they contain. The programming problems, or "tasks" ar§ described 

• in temis of the 'skills they use, and are .selected on the basis of this 

• description, relative to the . student's history of competence on each ;« 
skill. Figure 2 shows a simplified portion of the curriculuk network, ^ 
and demonstrates the relationship among the -tasks, skills, and techniques 

Essential among the' curriculum elements that descrijje each task are , 
its text, its skills, and its model solution. These elements, we feel, 
are also ^fundamental to the description of problems in m^ny technical 
curriculums, and '^re broadly applicable in areas unrelated to instruction 
in programming^ ' The optional elements in the task .descrip'tion. are also * 
us eltil ^categories' in other subject areas, with modifications specifically 
suited' io the given curriculum. 
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SKILLS 




Print 

string 

literal 



Print, 
string ' 
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Print 
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SINGLE 
VARIABLE 
READ & INPUT 



Assign 
numeric 
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with LET 
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string 
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with INPUT 
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Write a program that 
prints the string 
•WORSE" 



TASK HORSE 



Write a program that 
uses INPUT to get a 
string from the user 
and assign it to the 
variable Wl Print W$. 



TASK STRINGIN 



Write a program that 
first assigns the value 
6 to the variable N. . 
then prints the value 
of N. 



TASK LETNUMBER 



Figure 2; A simplified portion of the curriculum network. 



Computer programming, like most other technical subjects, is better 
•learned fhrough experience than through direct instruction, especially . 
if that experience can be paced at a speed suited to the individual 
student. Throughout the HEP course, the I)rimary emphasis is placed on ^ 
the solution of .problems presented in. the tasks. HEP does not present .., 
a sequence of instructional statements followed by questions.. IxTstea^^^. 
[a problem is described and -the student is 'expected to write his own V 
BASIC program to solve it. As he develops his BASIC program for ^eaeh 
task, the student is. directed^ to appropriate sections of the student 
mariual for -full explanations of BASIC statements, progyamniing structures, 
etc' He is also -encouraged" to use .the numerous student -oriented features 
such as an interactive debugging facility and' varions "help" options 
described in Section 

When a student enters the course he finds himself in tdsk '^GREENFLAG 

V ' • 4 ' 

Which requires a two-line J)rogram' solution. The .problem, as he is told, ' 

is- worked out in great detail In' the BIP student manual. Thus, the.. ■ 

trauma of being told to "write a program that' . . . "' in his first session 

. ■ ' ' ■ " 

is alleviated by. following the model dialogue, in which many typical 

^ ,• . . " ■ ' ■ ' ^ 

mistakes are illustrated, yet his hand^-on programming experience begins 

immediately. - * 

Figure 3 show:s all the curriculum elements, including the skills, • 
that describe each task. The text, states the r^cjuirements of the task 
to th^ student, ahd suggests any prerequisite' ifeading in the BIP student- 
manual. The hints (if any) present additional information at .the student 
request, and subtasks isolate a part of lfhe "main" problem as a smaller 
problem- which he is to solve, helping him reduce the main t'ask to, ' 



Skills pevelopetj 
inthelbsk 



Problem Text 



REPdhd Hints 



Ibsk.Whlch^ 
"Must Follow* 




Model Solution 
with Test Data 



"Sub lasks' 



"Disabled* Basic 
' Operators . 



Basic Operators 
Required in Solution 



Figure 3. Elements that describe a task. 
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separat^y soluble part's. The skills are' -the. specific progr^ing" . 
•.elements- required In the solution. The model solution is a BASJEC- pro- 
gram that' solves the problem -presented in^.the task, and is accessible 

' ■ _ V ' ■ \ , ■■ . 'I 

to the student if he* canriat r^ach His o^ solution. The. mode3r-also 
contains, coded tes*t injkUt data that is used to c6^t)are the results 

' * ' " It 

produced by the student's prog4:*am -against thoge of the model. The. must 
follow" tasks (if any) Vill follow tlje main task automatically, and' 
require, extensions of the student's, original solution. The -."required. '^^ 
operators" are BffilC statements, that must be- included in the student ^s. . 
program J)efor6 he 'iss allowed to progress ovbt of the current task j the /" 

.^Vdi^abled operators" are ^^TC statements '.tjaat, fpr pedagogical reasons,^ 

. ' ■■ . ■' . ■ ■ ■ . ■ » 

are not to be isrsed fei his solution p'rograii. 

'• * ' .■■ * ■ . ■ ' • ' ; *" ■ „ ■ 

Thefsequenfce of events that occur* §s the student works on a task 
'is shown/in Figure When he has finished the task by successfully 
running his program, the' student proceeds J^y requesting "MORE." His. 
progress is .evaluated after each task. In the "Post Task Interview" he' 
is asked -to indicate whether' pr ' not he needs^ more work on the skills 
required by' the - task, which are listed separately- for Kim. ^ 

As soon as the student completes GREEKFLAG, therefore-, fl^e instruc- 
tional program knows something about his^ own estimation 'of his abilities. 
■In addition^ for all ^future tasks his solution is evaluated (by means of 
.comparing its output with that of the model aolution run on the same test 
data") and the results ax^ stored with each skill required by the ^task. 
The program then has two measures of the student.I:S progress in each 
Skill-- his.self-evaluation and its own comparison-test ;results. 



^student 
progrjm 
•tails" 



Select and present 
task. 



Studem writes program to solve the problem. 
- BASIC Interpreter, ERR DOKTOR 



Hints 
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- Reterehces to BIP student manual 



Student runs program successfully. 
Ready to. continue, types M^RE 
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~% solution. Update- student histpry 
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After completing a task (he may of course- leave a task without . 
completing it) the student is free ^either to request another, or to work 
on^some^frograan^^ own. The. a^orithm by which BIP 



selects a next task, if the student ?requ^sts it.,^ is shown in Figure 5^^^ 
The selection process begins with the lowest (least complex) technique. , 
All the skills in that tdTchnique are put into^a set called MAY, which 
will become the get of skills that the next task ''may" use. 

The program then examines the student's history on each of the 
skills associated with the technic^ue, to see if it. needs further work., 
This crtterion judgment, is the heart of the task selectiojj^algorithm, 
an4 we have modified it often.. Two key counters in the history are • 

associated with each skill. One is base'd on the results of the solution 

■-^ . • • ' ., , . 

checker (described in Part I V), .and monitors ^ the student *s continuing 

success in- using; the skill. The other is based on his self-evaluation, 

and monitors his own continuing conf id^ncf^^jo^^ 

definition .of a "jnE^eds-work^^^ on which either counter is zero, 

indicating that the student- was- unable to pass the solution checker he 

last time that -skill was required in a task^ or that he requested more , 

work pn the skill the last time he^used it. Any such not yet mastered 

skills ard put into the MUST set. . Eventually the program will $eek to 

find. a task that uses some of .these "must". skills. - : 

If '.no Wch skills are . found (indicating that the student has mastered 

all the skilfs at that technique level); the search process moves up, by 
• ^ 

one techniqu^^ adding- all its skills to the,.MAY^ set, then seeking. MUST ' 
skills again. \ Once a -MUJST set is generated, the search be minates,. and 
all of the tasks are examined. Those considered as a. possible next task 
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Figure 5. Selecting the next task* 



23 . 

2^3 



for the student must (a) require at least one of the MUST skills, and 
(b). require no skills outside^ of the. MAY set.-N^inally, the task in this 
•group that requires the largest number of MUST skrU^s is- presented as 
the next task; Thus, in the simplified scheme .shown\n Figure 2, assuming 
that the student had not yet met th^ criterion on the skills^own, the 
~ftrs±^±^k^jfco_be presented would be HORSE, because its skill lies in the 
earliest technique, and would constitute the filiTlJUST^^setr— 5a?k — --r^->^ 
LETNUMHBR would be presented next> since its skills^ come .from the next 
higher technique;, STRINGIN would be presented last of these three^^^^j^^_^^_^ 

An interestin&^^GUr^^ has been developed 

hereT^If BEP'thas selected the MUST and* MAY sets, but-^cannot find a task . 
that ^ets the above requirements, then.it has found a Vhoie" in thp 
curriculum. Aftei* sending a message to the course authors describing 
the nature of the missing task (i.e., the MUST an^ MAY skills),' the task 
/Selection procedure, examines the next higher technique. "It generates 
'new, expanded MUST and MAY sets," and searches for an appropriate task. 
Again, if none is found, a new search begins, based on larger MUST and 
MAY sets. The only situation in which Jhis process finally fails to^ 
select - a task, occurs when the student has covered al^l of the curriculum. 

Our work for the coming year will concentrate on student models .that 
-'involve more than counter- type criterion judgments on the skills to be 
developed. We will attempt to characterize students* knowledge states 
and "difficulty" states explicitly by analyzing protocols. If we are 
•successful, a production system type of student model (Newell, 1973) can 
be used to guide the student through the curriculum material. 



•• IV. " HP^s Instructional Interpreter" and Interactive Graphic Features 

■ , ■ ... * ■• ' 

Most" of ETP's specially designed features are described by 'Barret 

' ,' . .. ' 

ai. (1^5). Since the .-publication of that report, a number o'f significant 
- improvements^ have been made to' existing structures, and a major graphic , 
instructional feature \Las been added. The purpose of ' this section is to 
present the motivation f&r these modifications and 1;o describe their 
.operation. 'Table 1 lists the ECP commands^a^^^ 

^----gmapeariojrth^ • • 

Improved Error Correction in the Interpreter / • 

• Because the HEP course is aimed at students with no previous pro- . 
graraming experience,, the error messages are-designed to. contain more 
infomation than that available' from "standard" BASIC systems, and they 
• are carefully worded in non-computer-oriented terns to avoid further 

confusion. . . 

In many cases, these expanded error messages appear to provide 

enough information to help students correct, their errors. However, 

' ' * . >■ . • «a ' . ' ■ 

especially in the case of the more naive' students, the generality of 

' . . ' ' ' ■ ■ ■ . ' * . ' I 

this error correction system proves, to be a drawback. Consequently, ; / 

. , the interpreter wa^ substantially modified .to provide more; specific 
infomation about the particular error the student hajs m^df?^ prime.ri] 
by identifying the place in his own line or t)rqgram at which BIP de, 
tected the error. • / ' 

Problems confronting the student . -.For the student learhing his 
first programming language the mechanical intolerance of the computer 
is often bewildering and* frustrating. The slightest spelling mistake . 
will cause the computer to behave in an une^rpected way. A statement 



Table 1 
hep's Student Command^-"-- 



II^IFORMATION 

Who (is. at this terminal) 
lJhaF~t^E^lc:'ainr^~-^tHg^ — — ^ 
Wheri (is it now) 
Gripe (to Stanford) ' 
Calculator 
Hardcopy : - " 



IWSTRUCTIOW ' ' 

' I. ' ' ' 

Task • 

_Mpre 

Reset (eSF'irilr~tHB'k5-)~^ — 
Enough (exit current task) 



PROBLEM SOLVING ' AIDS 



Rep 

Hint 

Subtask 

Demo 

Model 



DEBUGGING AIDS 



Flow 
Trace 



INTERPRETER COMMANDS 

Run % . . 

Scratch ., ^ • 

Se que nee C numbe r ii ne s ) 
List 

Edit^(.a line) • * , • 



FILE 'SYSTEM 

Files t^o see directory) 
Sa^e . 

'Get. • 
Merge ' 

Kill . . 
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that seems clear to the student- may be much less obvious to the coliputer, 
and often for, an obscure reason. One beginner, after successfully jenter- 
Ing the line . ' • 

into- her program^ to print out the answer, could not understand why the 
improvement i \ \ ' 

• ^ 10 PEIKT THE ANSWER IS X \ / .1 - 

would not worl^as-H&he-^had expect&di-^.Even more frustrating is the ' 
incomprehensibility of th^ computer's attempts at communication and 
clarification,' Error messages are fi-equently confusing or even mis- 
leading to the .novice. The error message - 

; .INVALID VARIABLE NAME . ' . ' 

may be triggered by an extra comma or extra quotation mark in the 
student's line, rather than, by a genuine error with a variable- name. 

Even a naive user is q^tfick to realize that a computer is not Intel- 
ligent, Consider the following exchange in which a student was trying 
to erase her working, program: 

■^SCRATCH " ' 

■ ''SCRATCH" '.IS NOT A VALID BIP COMMAND 

*WHAT IS A VALID BIP COMMAND? * * 

• • ' . ^™t is a. VALID BIP COMMAND?" IS NOT. A VALID BIP COtMAND 

Originally," BIP produced this Hind of conversational but absurd -response, 
which provides "little useful infoiroation. The student begins to dis- 
trust error- messages like this because the interpre"ter makes such 
obviously stupid mistakes, while 'pretending to produce intelligent 

natural language. Currently, BIP handles the student's input in a 

• • It ■ • " • 



more direct and '"honest" waj^,;^ as illustrated in the. sample dialogue in . 

■ \' ■ ' 

•;iSec€ion V, wi.lh appropriately mE?dest rape's sages like '-'YOUR PROGRAM DOESN'T 

SEESi TO SOLVE THE PROBLEM," 'and "^PBOR FOUND NEAR . / The student 

• ■ * ; •/ " \ ' . • ^ ...... .• , . . ' 

is not misled as to HCP's capabilities > and is thus encouraged to look ^ .'^ 

around the error indication ta find th^^ error itself. .\ 

Another difficulty common to new students i^ an inadequate or In- 

accurate conception of what the execution of their program enta1,ls* In 

most systems, variable assignment3, logical decisions > and programs 

branching are all' invisible to the user, and there is no way that the;'\ 

student can conveniently;, see the flow of execution of his program. 

Since learning to detug is a very important part of learning- to program, 

interactive graphic debugging systems are useful tools -that can greatly 

assist the . student's conceptualization of program execution • BIP makes.. 

available two. such facilities, .which have proved to be very, useful both 

to students and to' more experienced programmers. (These- debugging . 

facilities are. described under '^Graphic Features," later in this section,) 

Approaches to the prob3^ems of interaction . In .BIP's BASIC inter- 

/ - • . 

preter we^ have attempted to deal, with the problems inHerent in student- ; . 

computer communications.. Since' BIP runs in an interactive environment, . 
the student receives immediate feedback about his syntax errors, and 
information about other errors as soon as they are detected. . These, 
features keep the student frdm going too far down the wrong track with- 
out some, warning. " HE-P^s interpreter is built right- into the instructional 
program s,p that the- instructional system can continue to offer ^assistance 
after the .first error' message is given by the interpreter. . 



- EEP's error de-tection capabilities cover four different kinds of 
program; errors ; syntax and execution time errors., program structure 
errors detectable before execution but involving more.ithan the syntax 
of one line, and errors related to the- curriculum, task as signed^ making 
an otherwise correct program an unacceptable, solution to the problem. . 
Although it is not always possible to give an appropriate syntax error , 
message~^( syntactically wrong statements are by their very nature 
ambiguous)) we have tried to make BiP's error messages .as accurate as 
possible. In addition, we have added clarify4.ng messages for each error, 
including examples of aoryect and. incorrect, statements , which the student 
receives upon request. He may also'ask for a reference to the part of ^ 
the HEP manual that explains in detail the . statement he is trying to use* 

BTP uses a top-^^own parser to produce an internal code that can be- 
ef ficiently executed; The parser is used to detect syntax errors as 
well. -In*. addition to normal error checking, the parser specifically 
Idoks for certain very coMoa errors. For examp^, if the student types: ■ 

10 IF J r= 0 THEN GO TO 30 . ' * ; • 

an error- message inf orms- him that GOTO's are not allowed in IF statements. 
But 'lather than merely inform ..th^ student that his statement was incorrect 
"HEP pinpoints „ the student's error; . ^ . 
J , lo" IF 'J ^ 0 .THEN GO TO .50 : • • , . . 

SYNTAX ERROR: ^^GOTO" IN .IF STATEME.KT ^ • . - 
LINE NOT ACCEPTED (TYPE' ? FOR HELP) 

The f3jashing arrow' indicates/ the point at which the parser detected the 

error. Although the error message tells the student what is wrong, - it 

.' ■ * . 

does not tell him how to construct a correct IF statement. To get this 

. . .. . ■ ' • 29 / * . . ■ 



information he types a question mark, , and sees: ' •/ \ 

DON'T USE A "GOTO" "in AN ."IF" STATEMENT — THEY AREN'T^ ; 
I^GAL THERE ^ ' , 

■ TYPE ? FOR MORE HELP , ' . . " ' : ' ' . 

Alternatively,, thq student may type ?REF for .a. reference to the section • 

in the HEP manual containing a detailed* explanation of .IP statements. 

Similar information is available, following an execution error. 

Porexample, if the student ran the following program: 

■ • 10 DIM L (10) " . ■, . 

.20 FOR N = 1 TO 11 

30 L(N) = a ,. 

1+0 NEXT N • - . ■ . • , 

■ '» " 99 END 

he woiS^d^ receive a runtime error: 

• ■ -. ■ •' ■ , : ■. ^ , \ . ■ • 

•■ ■ ^ EXECUTION ERROR: LINE 30 , ' , " 

.INDEX FOR SUBSCRIPTED (LIST) VARIABLE OUT OF DECLARED BOUNDS 
. SUBSCRIPT, OF L IS -11. ■. • 

A question mark would elicit additional- infomatioii: . ; ; 

. POREXAMPLE .■■ \ ^ V \ 

^ ' ■ \ - 10 ; DIM •X( 20) 

-- *20--J---- 23 - ' - ^ ' ■ - ■ ■ ■-' ' 

30. X '(J) 0 " ' ■ ' ■ 

TS INCORRECT/ BECAtJSE 25 IS GREATER THAN- THE DIMENSION OP X 
In addition to syntax and execution time errors, there are some- 
program structure illegalities "th^it can be detected before execution. 
(strictly speaking, these are syntax\errors , but they inyolve more than 
one line^o^ code and eire not 'generally d^alt with by BASIC interpreters.) 
We )\ave found that identifying these . struofe^ral bugs, rather than letting 
them appeSr as execution errors ^ eventually, ca^ often clarify the error 
for the student; The* ERR DQKTOR routine is calleH^ by the RUN procedure, 
and .-examines the .program before attei^grting .execu.tionV It notifies the 



student of errors like branches to ,npn-e,:?cl5tent lines, branches from a . • 
Iftae to itself, illegally nested FOR. .l^ffiJJT^^^ improper transfers 

into and out of ^ subroutines • ^s,^^ ■ • " 

J 'Since the -BIP coufse runs without' human graders ,^"^simple "solutioii 
checker" is built in to evaluate' the .correctness of his prbgram- It' 
works by running the model solution and, comparing its output . to^> the out- 
put of the student's prqgram. While this, appro.ach does not eatch all ; 
possible bugs in a student «s ; program, it is- similar to methods U|>ed by 
most h^aman instructors in grading programming assignments, and. executes 

very quickly on our system. ... - ' - • • 

The. solution checker executes the stored ^model solution invisibly,- 
.storing its output, then executes the student's program in the t^ame way. 
Each time the student's program produces output, if is . compared to the 
list s'tored from, the' execution Qf the model, and any matching elerr;ent - 
in that list is flagged. If any unflagg-ed output remains - in the list 
when execution is- cumpleted, the student fstold that his program 
'^ddesnn. seem to. ooXve -the problerr.. " If all outputs of the ciodal liav^ • 
been- ..matched, he ii told -that his prCgrari, "!■ "la; .;k. 

• . Because n.arty of I'tP*3 tasks require it.'t<f;raetj.Ye programs tliut deal' 
-with a hj-pothetical usea , tnfe solution ehecker ir;u$t be -able u perfor-'r; > 

.its comparison-by-execution on suitable "test yalueB-, thoB,e tnat n.lght . 

be' given by the use in response .to au IWPUT statement irr the program. 

These values .are stored with the model «olutiou as part of tlie. curricnluiii 

.and are chosen to. provide a test of the stuuenfs program appropfialfe te 
-'the level of difficulty of the task and to ttie explicit- i-equire'iieritu ,• . 

stated in 'the text, . ' ' ' 



'When HEP executes the student*s program, it asks* him to provide the 
-names of the variables he used for the INPUT functions required j- an 
example might be "What variable do you use for the u^r's first number?" 
(The description bf the variable's function is also stored in the ra^odel, 
as a REM statement that gives- infomation, but does not affect execution - 
in any way.) Then the solution checker assigns the test yalues to the 
student's own variables, • allowing it to execute- his program on exactly 
the same input as was used' in executing the model. .If the student • 
program is found to be inadequecte, he is advised to run the DEMO to see 
the model in aqtion. For interactive programs, the test values are 
suggested as appropriate input to the DEMp so. that he can see how the 
model ' solution handkes tho^e values. Frequently, the student 's- program 
will fail to deal w^th the test values, and the failure , is evident from 
the correct example; provided by the'EEMO. In these cases, the solution 
Checker gives iristnictiye help in Just: those areas that cannot be iden- 
tified by the other ierror detection facilities. a" 

The solution ^liecker ignores output of string constants, since they 
are* frequently optibi^al messages oriented toward the hypothetical us^r 
of the student's program, and rarely affect the real correctness of the 
program. Extraneous Wtput (i.e., output beyonajhat produced by the 
model) is also ignoredv for much the same reasons. However, in those ;- 
tasks where string constants or "extra" output are considered relevant 
to the correctness of the program, the focus of the sblutio'n checker 
.can be explicitly narrowed by additional infomatt.on coded and stored 
' along" with the test values.. • . 



'Though simple and obviously limited in some ways, BIP's solvation J^^^; . 
checker is an effective tool, not only, in acting as a ^rade'^:' that 
governs progress through the curriculum, but also as. an additional source 
.of instructive infoiroation.; Particularly;? in the more demanding inter-," ^ 
. active tasks, the values used by the checker and suggested, to the student 
"add to th$ b^neficial'learning effects of hands-on experience and mariip^^ 
:Ulation of his own BASIC programs../ . . _ . 

New Interactive Graphic Features ' • ^ *. 

: . ' In Addition to the instructive capabilities built into the inter- 
preter, BIP offers two rfecilities that use the CRT display screen as- d 
dynamic source Of infoiroation.' The * REP command presents a flow chart- 
like representation of. the model solution for each problem that can be 
expanded at the student's re^ueqt to reve.al more infoi^ation about the * 
model's programming structure. The FLOW command is a debugging aid that 
allows the student to execute his, own prograrn a line at a time, and makes . 
•'each ptep of the execution fully, visible. v 

REP: .A grk^hic problem solving aid . Several typfes of "help" infor- 
mation are stored with each task as part of the curriculum network. .Most 
' of the commands by which tbe.4stud.ent accesses -this information are 

c ■ . * ' 

■ ■ ■ • . ■ . ' y 

discussed in "Barr'et al. (1975')/ and only a brief description is included 
here, to provide the 'context for the description of the flow-ohart assis- • 

. trance system, .REP. 

HIKTs present additional infoiroation about the requirements of " 
■ tke taslj^or suggesldk^^ the student construct part/.of the . * 

solution program. For the morT^cOi^O^Qated tasks, the student may , '^^ 
request a SUBttask, which presents a smaller^^^rt^qf^he .problem, to be 
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solved . separately and integrated with the larger "main" task.' The EEMO 
executes the . model solution, as" an' illustratiori (rathe^,than a descrli^tion) 
of the requirements of the taskj it is particularly ^isenil in the inter- 
active tasks in showing the student how his program . should interface with 
the hypothetical user.. ^ ' ^ 

Here we describe REP, the new graphic problem solviTig *ai]^ in some 

• ■ . ' • • . •■ ^ ' • ■ 

detail) using a specific example from tlije curriculum. The requirements 

. ** " . ..I ■ *^ . 

of the sample task are: , ' / 

MtdTE A PROGRAM THAT USES ONLY ONE PRIKT STATEMENT A?ID POINTS THIS PATTERN. 



The solution is short but the task ±s difficult:, ^ 

' • • • .X. ° * ■ ■ . ■ . ^ ■ ' ' 

10 s$ = ■ ■■ • 

20 FOR- I =ia'»TO 8 '. • . ■ , . . ■ , 

' I 30 PRTKT S$ . : • ■ . 

. W s$' = ". " & s$ •■ ■ ■ ' ' . 

. 50 NEXT 1 , ' • ■. 

99 END, ■ . » 

The •critical .points of the task .are that S$ must , be initialized to 
the PRINT statement must. precede the re-assignment within the loop, and 
the variable S$ muqt be given the correct concatenation (a space appended 
to. the beginning of the strinjg) '^^^f ■ * . . s 

Figures 6a through 6c illustrate the information available to the 
student via 'REP. The display on the CRT screen is considerably eas'ier 
to read than this static illustration, and the jnovemelat of the display 



1. The, student"^^ types REB and the screen displays 
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A: INITIALIZATION 



B: PRINT A # AND MAKE PREPARATIONS 
FOR NEXT PR.INT -■ 



[C:] GO BACK. AMD PRINT AGAIN 



To get more detailed information about the contr'ol structure, 
the student tJypes C: 
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A: INITJALIZATION 



B: PRINT A- # AND MAKE PREPARATIONS 
FOR NEXT PRINT 



3- 



.[CM/QO BACK AND PRINT AGAIjl 



I'" 



Figure 6a ^ .The REP display 
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3. NoteHhat'b Is still available for deeper probing In the above 
display. The "student 'types B: 



• A * 
««««■•• 



»»»»| 



• ■ B2 • 



««««+ 



A: INITIALIZATION 



BK 30 PRINT S$ 




B2: CONCATENATE A SPACE TO S$ SUCH THAT 
THE NEXT # TO BE PRINTED IS MOVED 

Over one SPACE , 



[C:] GQ^CK AUD PRINT AGAIN 



i», .The original output, box b)as been expanded. The student types B2: 

A:,, INITfALlZATION 



* A 




\ ♦<««« 



\ • . 



B1 30 PRINT S$ 



B2 10 S$ = " " & S$ 



[C:l GO BACK AND PRINT AGAIN 



Figure 6b. The REP display (cont'd). 



5» The student tyt>es C again to see how the loop Is carried out: 

, ■ ' ' , '^.* 

» A » A: INITIALIZATiON 

' ^ +»»»»! C 20 FOR I = 1 TO 8 

t ' ■ ■ I . • •■ 

I ««««««• 

T • » Bl » Bl 30 PRINT S$ 

I" I" ■ 

I ' ««««««« 

- T ' » B2 » B2. ,40 S$ = " " & S$ 

^-r • I' ■ ' ■ . 

+««««+ 

C 50 NEXT I 

. 99 END ^^ 



, Ml 



Figure 6c. The REP display (cont'd)/ 
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allows the student to see the direction of its "growth." Symbols shown 
in square brackets represent the control structure of the program and • 
blink on and' off tjD fOcus the student's attention oh their importance. 



. In using HEP- 



the student, is allowed to probg the representation in 



both lire ad th and 4^pth ^nd in any sequence. If he probes in the breadth 
dimension he ;may first look at control structure information and find , 
that the program requires a loop. Next he may look at IKPUT/OUTPUT or 
other key information. Thus, oilqe he has established that the program 
requires a loop, more information might l^e requested on control structure 
until, finally, he is -shown the actual BASIC code. The implententation 
of REP allows us to esqperiment 'With various aspects of its operation; 
flags can be set to control which labels will blink, how much- infomation 
will be displayed whether or not .EEP' itself is aVaiiaible to a given 
student, etc. 

FLOW; A debugging "^facility ;' We ■ have linplement^qL ;^wo^jt:racing facil- 
ities 'to assist the student in conceptualising the e^^cutlon of his 
program. Tracing a program is difficult to do correctly by hand, since ♦ 
one tends to make the same mistakes over and ove^r.. It is especially 
difficult "for. beginning programmers, who may hot underctand the function 
of some statements. .BIP*s TRACE option automates this process. It 
allows the student^ to see exactly how his program is executing, and to 
identify the point at which the program begins to stray from -what he 
intended. As each line of his pro'gram executes, the line number is ^ 
displayed on his teletype* or display terminal. Any variable- assignments 
perfo-med in that line are .also indicated, as» well as any input or output 



PLOW is a more sophisticated program tracing aid designed for CRT 
displays. The main program is displayed on the terminal, with the text- 

of all subroutines removed. Each 4:ime the student prtsses the CR key, 

r • a,. ' , . . . , 

o^ie line of his program is executed^ and its line number blinks'>on the ' 
sci^feen display* When an IF. or ^ GOTO statement is ^^ecuted, an arrow is 
drawn on the screen to indicate the transfer of control. 

' When a subroutine 'is called, the main program display is replaced 
by the lines that make up the subroutine,- Additionally, a message in 
the comer of the ^careen indicates the level of nested subroutines. 

T!he student may Si Is o reqtjest that up to six. variables be traced.. 
The current values of all trac/ed variables are showEu at the top of the- 
screen. If an array is traced, the value of the most recently assigned 
array element is- sho>7n. 1 

^'^ ■ ■■■ v-j* '-^ ' ' ' . ■ 

; .The stuffent may also speci|y a line number in the flow command, . 
The program will execute continuously 'without waitijig for the student 
to .press the key,: until the specif led' line is reached. At that point, * 
the prqgram will' resume s tep- by- st^p- execution. This feature allows the 

■student to reach the troublesome part of his program quickly.. 

Figures 7a through ye illustrate a hypothetical FLOW through a 
simple program with a FOR. .NEXT "loop.. Each figure shows, the progress 

,of the execution triggered .by ;the student's key press. Arrows are used 
instead of blinking line numbers to indicate the current line and any 
changes caused by its execu^on. The changing value of the^ variable I 
is. traced in the upper left comer, while output appears below the 
prograia listing. If this- were an interactive program, input .would-be . 
shown in. the same area. * • 



•FLOW I . 



Is.?, MAIN PROGRAM 

HIT <CR> TO 
RUN 

> 10 PRINT "START" , " ■ 

■20 FOR I = 1 TO 3 
30 PRINT I . 

»»0 IF I = 2 THEN 100, ^ ' . 

<. . 50 PRINT "NOT . TWO" 

100 NEXT I- : > 

999 END 



— > OUTPUT: START 



/ 



I <_— 



10 PRINT "START" • 
.> 20 FOR I =1 TO 3 
30 PRINT I 

40 IF I = 2 THEN 100 
' 50 PRINT '"NOT TWO" 
100. NEXT I 
- 999 END . . 



I 1 



10 PRINT "START" 
.20 FOR I = 1 TO 3 ■ 
— > 30 PRINT I 

H« IF I = 2 THEN 100 
50 PRINT "NOT TWO"- 
100 NEXT I 
999 END 



---> OUTPUT: *1 



Figure 7a. The FLOW display. 



1+0' 



I 



10 PRINT "START" 
20 FOR I =1 TO 3 
30 PRINT I 

HO. IF I '= 2 THEN too 
50 PRINT "NOT TWO" 
■ . 100 NEXT I 
999 END 



■3" 



I = 1 



10 PRINT "START" 
20 FOR I = 1 TO 3 
30 PRINT I 

.i»0 IF I = 2 THEN 100 
-w> 50 PRINT. "NOT TWO" . 
10Q NEXT I 
999 END : 



— > OUTPUT: NOT. TWO 



I = 2 < 



' 10 PRINT "START" 
20 FOR I = 1 TO 3 . 
30 PRINT I 

40 IF I = 2 THEN" 100 
50 PRINT "NOT TWO" 
— -> 100 NEX,T I 
999 END 



Figure 7b. The FLOW display (cont'd). 



ERIC 



10 PRINT "START" , 
20 FOR I = 1 TO 3 

+— > 30 PRINT I 

! ■ 40 IF I s 2 THEN 100 

! . 50 PRINT "NOT TWO" 

+ 100 NEXT I 

999 END , 



i— > OUTPUT: 



1 = 2 



10 PRINT "START" 
20 FOR I .= 1 TO 3 
J 30 PRINT 1 

> HO IF I = 2 THEN 100 

50 PRINT "NOT TWO" 
100 NEXT I 
999 END 



3 < — 



10 PRINT "START" . ). 
„ 20^ FOR I = 1 TO 3 ( 
" 30 PRINT I I 

+ i|0 IF J s 2 THEN 100 

! 50 PRINT "NOT TWO" 

+ > 100 NEXT I 

999 END 



Figure 7c. The FLOW display (cont'd), 



ft I 



10 PRINT "START" 
20 FO)l l a 1 TO 3 

+ > 30 PRINT I 

! ' MO IF I = 2 THEN 100 
! 50 PRIUT "NOT TWO" ' • 

+ 100 NEXT I ■ 

999 END 



> OUTPUT: 3 




10 PRINT "START" 
20 FOR I = 1 TO 3 
30 PRINT I ■ ■ ^ 

r— > 40 IF I = 2 THEN 100 
50 PRINT "NOT TWO" 
.100 NEXT i 
999 END . , 



10 PRINT "START" 
20. for' I = 1 TO 3 
30 PRINT I ; 
40 IF I = 2 THEN 100 

' > 50 PRINT "NOT TWO" ■ 

100 NEXT I 
* 999 END 



> OUTPUT: NOT TWO 




ERIC 



igure 76, The FLOW display (cont'd), 



k3 4B 



I s 1» < 



10 PRINT "START" 
, 20 FOR I s - 1 TO 3 • 
30 PRINT I . 
UO IF I = 2 THEN 100 
'50 PRINT "NOT TWO" 
100 NEXT I 
■ 999 END 



I = U 



\ . ' 10 PRINT "START" 

20 FOR I J? 1 TO 3 
30 PRINT i 
. ' . . HO IF 1 = 2 THEN 100 
50 PRINT "NOT TWO"" 
100 NEXT I 
>. 999 END 



EXECUTION COHfLETED AT LINE.,999 



\ 



Figure 7e , The FLOW display ( cont 'd ) , 
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V. An Annotated Dialogue 

• ^ ' • o: -- • . ..• . • ...^ 

HIP is a. programming laboratqzy using the BASIC language., It pre- 
'■ " ,. • • ' ■ ' ' ^ . - 

sents an individually Sequenced set of problems that the student is to 

solve hy writing BASIC programs. Typically., the tasks ' Joresent very little 

instruction in the syntax of BASIC; a manual, describing the language and 

the BIP system is 'provided for reference and detailed infomation. The . * 



task shot/n here is clearajr not one 'of the first that the student voiild 
see/ since it requires some ^kill both in planning the program's structure 
and in* using appropriate BASIC construct^, to carry^.out the Job. 

The student/BIP -interaction i-s shown on the left, with all student 
input preceded by an asterisk, {^y* Our descriptive and explanatory f 
annotation is indented to thg right. 



The student signs on to the system. 



WELCOJ^ TO BIP 11 
TYPE "?\ FOR HELPc" 

A. 



-Some sort of 'asJ^l^^tance is always available. 
If the student types a question mark, BTP'^ 
response is related to the most recent err®r, 
as will be seen below. 

Wh^ri' the stude^ requests a problem, BIP examines 
hig hiBtoiy and\ries^ to selegt an appropriate ^> 
one'. Most' tasks^are interactive, requiring the 
student to 'get input from^a hypothetical "user." 
Thig one isn't, but students have found it : 
challenging and interesting; 



^TASK' 



BEGIJf TASK: XMAS . \ , 

©N THE FIRST DAY OF CHRISTMAS, SOMEONE »S TRUE. LOVE SENT HIH/HER . 
A PARTRIDGE IN A PEAR TREE (\nE GIFT ON THE FIRST . DAY) .. 
ON THE SECOND. DAY, THE TRUE SENT TWO TURTLE DOVES IN ADDITION TO " 

ANOTHER PARTRIDGE (THREE GIFTS \ ON THE SECX)ND DAY). ' \ 

THIS CONTINUED THROUGH* THE 12TH\dAY, WHEN THE TRUE LOVE SEIWT 12 LORDS, 
11 LADIES, 10 DRUMMERS, . • .. AL]\ THE WAY TO YET ANOTHER PARTRliXJE. 
' .• ' . * - «> ■ \ • 




WfilTE A PROGRiM THAT COMPUTES AND PRINTS. THE NUMBER OF GIFTS SENT ON 
' THAT I'WELFTH DAY. ( THIS IS NOT THE SAME AS THE TOT/\L NUMBER OF GIFTS . 
SENT THROUGHOUT AIL. 12 DAYS ~ JU^T THE NUMBER SENT ON THAT- SINGIE J2TH , 

DAY,.) . . » ■' ; 

' This is a faiVly difficult problem, and 
, ,^ ' several problem solving" aids are available, 

. . c • In the previous section we discussed the 

• REP feature which helps students break down • 
-problems with flow charts^ (Its, ; graphic 
nature cannot be demonstrated here.) 

. ' Additional text material is also" available, . 
0 . either as a clarification of the task itself 

'* ' V or, as in this case, to help the student 

construct his own algorithnf^ 

*HINT ' : 

HINT #1' . , / 

TECS PROGRAM REQUIRES A LOOP, EACH EXECUTION OF THE LOOP INVOLVES 
ACCUMULATING THE VALUE OF THE INEEX INTO A TOTAL/ 

*HIHT ^ ^ ^ . • ' 

HINT #2. ■ ' ' ' 

FINDING A .TOTAL OR SUM ALMOST ALWAYS MEANS TWO THINGS: . 

1, SETTING. A VARIABLE EQUAL TO ZERO OUTSIDE h LOOP, 

2. ACCUMULATING VALUES INTO THAT -VARIABLE WITHIN THE LOOP. IN WORDS, 
•TOTAL EQUALS OLD . TOTAL f>LUS ANOTHER VALUE, 

- * 

Jn some cases, the task involves^ a number of - 
• ' complications, someof Vhich can be isolated , 
and presented 'as smaller "subljasks as shown 
■ ^ - ^ here* This' subtask directs the student's 

. ^attention to the prob^^em of wilting a simple. 
: loop using a FOR. .NEXT statement. The studfnt 
. ; • always has .the option of completing the subta'sk 
■ or of returning io the "main" task as soon as 
_ . / he has gained ^the , information he thinks he ' 

needs to solve the o^riginal problem; 

*SUB 

■ ■ , ' "'• ' * ■ ' ■. ' ■ ■ ' 

BEGIN TASK: S-:FDRLOOP • • 

WRITE A PROG.RAM WITH A' LITTLE ■ -FOR . , NEXT- LOOP. 

THE. WORK OF THE LOOP IS JUST TO PRINT THE VALUE OF THE LOOP'S INDEX 
(the VARlABIrE THAT KEEPS. TRACK OiF.THE NUMBER OF TIMES THE L'OOP HAS ' \ . * 
BEEN EXECUTED). ' ' ' ' , . 



WHEN YOU RUN THE PROGRAM,' IT SHOULD Lok *f.IKE IT IS COUNTING FROM 
.1 TO 10. • ' ^ ' 

9 * The subt^sk' is treated as a regular ^ask by. . 

. V ' • BIP, and can have hints' and subtasks of its 

• » : 'own. Here the student malces several. attempts . 

' ^« • . ■ at a solution, illustrating the el'ror*'. detection 

r. facilities.^, ' , ^ 

' ' Fir^t, although the problem specifically * 

' ' ■ ' required a FOR. .NEXT loop^ the student will 

try to construct a* "hand-made" loop of his owo. 

" • ■ ■ ^ ■ ^ ■ . . . 

*10 LET I =^1^ " ' ■ . 

^20''praNT "^THE INDEX ISj I . ' \ 

SYNTAX ERROR: UNMATCHED QUOTE MARKS r- FOUND NEAR "THE INDEXsTS' 

LINE NOT ACCEPTED (TYPE ? FOR HELP) \' ■ ' 

. . ' V " ' \ -' ■' 
HP(s inteipreter was specially wi^tte^ 

pro^kde as much inst^riction ai^d errpr 
correction as possible, in contrast $o 
' "standard" BAjSIC systems. , H^'rWvthe student 

has made a common error by o«^it1^ing . th^ , 

V • closing quote- marks 'in his I^iNTVstatement. 

BIP is able to^ indicate the point vat wl:\l/:h- ^ 
' * it detected the efror, and' to give\ furtlier ,» 

infomation specifically related to. thi4 
• particular incorrect BASIC s.tatement: 



^9 



•"3?HE INDPX IS*' HAS AN ODD NUMBER OF QUOTE MARKS 

REMEMBER THAT ALL STRINGS MUST. HAVE .A QUOTE AT THE BEGINNING AND END 



TYPE ? FOR MORE HELP 
^?REF 



II.7, ir.8 



BIP can always present Vn appropriate manual 
reference for all errors. 



4- 



This student gives up tHe fancy PRINT statement 
and includes a^ simpler'.but adequate one. • ' 
Unfortunately/ he is soin to find that his 
loop construction is unacceptable* for this 
taskt • - 



S3 



*20 PRINT I • 



*30 IFI = 9 THEN, 99 - ; \ 
FOR THE PtTRPdSES OF THIS TASK; ,DO]J 



USE THE 'IF' STATEMENT, 



*SCR ' . , 
*10 PRINT 1 
'.*20 PRINT 2 
*3b PRINT 3 



His'dilgri^l plan for a Ipcp including ^ 
"conditional branch will have; to, be abandoned. 
Undaunted, -<the" student now triejs to solve 
the ^)rQblem Vwithout ti loop .' Ffe * first delfe-tfes 
'the lines 'he\ has 'entered so . far. >- 



*LIST 



, After entering all of the PKCNT statements, 
he lists his plrogram, and then runs it. 



10 PRINT 1 
. 20 PRINT 2 " - ' 

"30 PjJlINT 3 ' , ' ■ ■ ■ ■ ■ S 

UO PRINT W 
. 50 PRINT, 5 • ■ ■ " ' 

60 PRINT 6 . . . 

• 70 PRINT i' ' ' 

80 PRINT 8 • . ■ ' . , ' 

90 PRINT 9 • 

99 iEND • 

; . ' . ■ • ., • ■ 

*RUW . ' 
EXECUTION OF YOUR PROGRAM ' 



EXECUTIOU COMPLETED AT LIKE 99 



The program' runs vi^hout* execution errprs . 
When- the student considers -his projgrara to be . 
an adequate sclutioh to the problem posed in . • 
the task,-?* he requests BIP Is evaluation of it ' 
Igr typing MORE, indicating that he vHnts -to ' 
proceed to another task. Here, thd student 
seems to^ be satisfied, ,and decides to co^itinue - 



mow ^ ■ . * V 

WAIT. . SOMETHING IS MISSING. FOR THIS": TASK, YOUR PROGRAM SHOULD ALSO ■ , 
INCLUDE' THE F0l1X)WING ' BASIC §TATEMENT{S.) : ■ . • 

FOR*'-; . • ' , • ; '. 

ADD TO YOUR program' AND -RUN- 'iT AGAIN BEFOpi ,YOU TYPE 'KORE ' 

k 'r f -- 0 ■• 

' ■ . ' : ■ - 

^ ' ' . BIP has recogniz^ed that the es&ential FOR..^. 

/ , . KEXT loop is missing from the student's program, 
* -and refuses to evaluate' it further*; Finally, ; 

he' sees the lig^'t, deleting his last effort 
^ .'•[ acid trying again, this time with a FOR. .NEXT . • 

^' . ■ * " loop# - - 



*SCR 



*10- FOR I =^ 1 TO 9'; 



*.20 PRIHT T 



5^99* END 



V 



Th^s three-line progr^-i^ of " course 
syntactically indorrect/ as_tl;ie re is an open 
loop. BIP has. a routine called ERR DOKTOR 
vhieb checks the structure of complete programs 
before they aire run, so that such errprs dp not 
appear as less infomative execution time errors. 



*RUN 



ERR DOKTOR DIAG^NOSIS: . 'FOR' STATEMEIW WITHOUT MATCHING 'NEXT! 



LOOK AT LINE NUMBER: •- 10 
TYPE ? FOR HELP ■ 



As always, .more help is available 'after the 
original error message. •. 
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YOU ARE MISSm THE 'NEXT STAipiENT C0RRESP0NDIIJ3 TO THE ' -FpR- ON 
Vthis LIHE. *' " » - ■ 



Type. ? for more help 



After- listing his prognsta, the student .sees 
and corifects the loop error, and the progrip 
executes successfulljr. 



- *LIST 



lO fOR I = 1 TO .9 
■20 FEINT I ■ ■ ■ 

99 END- • ' 

.*30 NEXT, I ■ - ■ 

*EUW . ■ - . .■ 

EXECUTION OF YOUR PROGRAM 



i: 
■3 

6" ■ . ■ - , ■ ' ■ ■ 

7 ■ : ■ 

a • ' 

'9 ;• ■ -. 

EXECUTION. COMPLETED' AT LINE 99' 



Again, the .student is ready for MORE.- HEP 
evaluates- his Solution l^y invisibly executing 
both the model solution and hi's. program on . 
the same "test" data, then ^omt)aring their 
output, in this ciase, the model produced 
output different from that of .-^he student's 
program, so he is told that hi q program doesn't 
seem correct, H6 elects to remain in the task 
to try again. \ 



•h-MORE'. , 

EXECUTlbN OF THE MODEL 
■ EXECUTION OF YOUT PROGRAM 
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YOUR PROGRAM DOESN'T SEEM TO SOLVE THE- PROBLEM. . , _ ' . . . 

IP YOU ARE COMPLETELY COKFUSEp OR LOST, YOU CAN LEAVE THIS TASK • 
RIGHT NOW. ' ' ■ 

DO YOU WANT TO STAY IN THE- TASK? TYPE • Y . OR U: Y • . 

. O.k. TRY AGAIN. YOU. MIGHT LOOK [AT A -DEMO- OF THE: MODEL" SOLUTION 
AND COMPARE. ITS, OUTPUT TO THE OUTRUT OF YOUR PROGRAM. 

Many -of EEP's tasks are, interactive ; the 
. . * student's program must be able to ask questions 

.1 ; • . • of a .hypothetical' user, and to deal.vith the 

user'^s input in appropriate ways.. The, DEtlO 
^ ' . . ^ * • ^ feature .allows the student to execute the 
. ; ' ^ ' stored model solution to see the kind of 

interaction required. In this task, the demo / 
' ^ ' ^rves to remind the st^^ident ' that his program 

• . was supposed .to covint to 10, not 9« 



^DEMO . 

EXECUTION OF THE MODEL 

■.THE INDEX :IS- 1 - " . ■ ; 
THE INDEX 'is 2 

THE IlTOEX iS 3 . • • 

THE INDEX IS k. ■ " ' 

THE INEEX IS 5 

THE INDEX IS 6 , 

THE INDEX IS 7 • ■ • 

THE INDEX!- IS . 8 ' , - 

THE INDEX IS 9 

THE INDE}^ IS 10 

■WE ARE -NOW OUT OF 'THE. LOOP. .-. 

EXECtJTi;Q,i^ CaMPLETED.AT LINE 99 
DEMO llNDED • ' 



And he corrects the first line. BIP reminds 
the student that, the line has been, changed. 



^10 FOR I = 1 TO 10 
WAS : IQ FOP. I - 1 TO 



EXECUTION. OF YOUR PROGRAM . 



EXECUTION COMPLETED AT LIKE 99 ' 

J Again-, satisfieoi, '-the student asks for MORE. 

*MORE ■ . , . •. .". . • 

EXECUTION OF THE MOEEL » ' ' 

EXECUTION "SlF YOUR PROGRAM 

LOCKS CK . 

■ ' "Following the successful comparison, EEP 

presents the "post task interview, "'in which 
• ' „. the student's own evaluation of his under- - 

.standing and skills is acquired and stolid. 

ft V ' ' . ' ■ ■ , 

■ - Fitst the model solution is displayed, and the 
student is asked whether or not he understands 
how. it works. ' ^ , 

POST TASK INTERVIEW ' * \ 

: . ' • ' ' . I 

HERE IS A TYPICAL SOLUTION TO THIS TASK: 
10 FOR I = .1. TO 10 . 
20 PRINT "THE INEEX IS "j'l 
■30 NEXT I ■" ' 'i 

l+O-PKENT "WE ARE WOW d,UT OF THE LOOP." 
99 END ' 

' " ' ■ ' ■ • » . 

K) YOU- UNDERSTAND THE SOLUTION? ■ (y/N) : Y 

Then^ for eaph'of the specific skiils required 
* in the construction of .the program for this 
task^ the student is asked to indicate' whether 
, or not he feels confident in his u^e of that 

skill. These questions are a major source of 
infomation for ECEP's task selection proce^ss, 
* .. by which the next' ^task will be chosen at . a 

. ' ■ . level appropriate f^or this student. "Those' 

skills to which the student responds *'y" will 



not be sought in his next task, since he feels . 
• that he has had enough work on them. , Those to 
, ^ " which he responds • "n", on the other hand, will 

be looked for explicitly. Unless he has 
.'..^ ' ; . exhausted a portion of the curriculum, some of 

^hr^ . those "n" skills will defiriit.ely be required in • 

XtV . his next task*, providing him with the opportunity 

■ ' ; ' to use those -skills ^ again, in a new context. 

THIWK ABOUT THE SKILLS USED IN .THIS"TAS1<., FOR EACH SKILL, . 
TYPE Y IF YOU HAVE HAD ENOUGH WOHC WITH THAT SKILL. 
"TYPE N IF YOU THIM^ YOU .NEED MORE WORK ON IT. 

FOR . . NEXT LOOPS WITH LITERAL AS I^INAL VALUE OF INDEX : Y 
MULTIPLE PRINT [STRING LITERAL, NUMERIC VARIABLE] : N 

. . Since, he did not use .the "multiple print" 
statement shown in line 20 of the model, Our 
' " student indicates that that ykill wmld be 

appropriate in hia^next problem. 

EEP informs him that he has returned to the 
larger 'task at hand, and allows him. to have 
• ' its text re-displayed. . " 

RETURNING FRpM A -SUB TASK,. ' / • 

YOU ARR^IN TASK XMAS. . . - ' ■ 

DO YOU ViALMT THE TEXT PRINTED OUT?, .;T'kTE Y OR N. . 



VI. An Experime-nli Evaluating EEPVs Individualizati^on Scheme - 

" * ■ ' ' '■. • ' ' ■ ' 

An experiment comparing the effectiveness of ETP^s teaching strategy 

with a more traditional "Branching" strategy va& ixiH Ih February and 

March, 1975. ETP's-task selection .strategy is described in detail in 

Section TII. The control group followed a predetermined bra.nching /, 

strategy through the curriculum arrived at as f ol,lows : A committee of 

staff members, with experience both in" curriculum design and in teaching , 

programming orde'red the .tasks by- complexity and the inherent hierarchy 

■ • ' ■ . 'U . 

of programming concepits required for their solution. For each task, two 
"next^* tasks were- specified, one to be presented if the student' sudcess- 
fully solved the 'current task without seeing . the model solution, and the 
Other to be. presented if he failed. All of the existing EEP tasks w6re 
incorporated'' into this fixed path, as either "njain line" or .remedial, 
problems*. * . 

Fortv-two Stanford students, 22 meh and 205:^men, we're recruited 
cib .subjects for the experiment. All were given the Computer Prograraminj 
•Aptitude Batteiy ,(196U) as a pretest. Two matched groups, eaqh wit,h 11 
men and 10 women, were created by ranking the subjects » pretest scores, 
and. alternate;]^ assigning subjects to groups. 

Subjects yorked at CRT terminals for 10 one-hour sessions., signing 
up for each hour one or two days in advance. Of the original U2 subjects 
who 'took tfife pretest, one woman failed to begin work on the course; the' 
other hi all completed the 10 hours work within three weeks and then • 
took an off-line posttest. . , * 

-The^strategy for selecting the "next task" was the only, difference , 
in treatment between the groups. Since this prqpess was invisible to 



the students, their interactions with HEP appeared identical. They had 
access to all of the ' interpreter and assistance features,, and both groups^ 
were given the post task interview, after each task, although the infoma-^ 
tion collected therfe was not used in the task selection decisions for the 
fixed path group. • 

Extensive 'data vete collected on-line as the subjects worked on the 

. , " . " -i-f ' ' ^ .... 

couri^e, including complete .protocols whose analysis we ^are now attempting 
to automate for our research on student modell. For purposes, of comparing 
the'perfomance of -the two treatment groups, the infoimation recorded for 
$a^htask is most interesting/ This information includes: 

- whether the subject "passed" the feolution checker on- this task 

- whether he passed the checker on -his first. 'attempt 

- whether he. said he understood the model solution in his PTI 
-"^whether he requested and saw the model solution before completing - 

the task * 

In addition, a cofeprehenslve posttest was administered off-line.. The 
test. was«designed to measure the students* ability, to- interpret correct 
BASIC programs, .to complete specified sections of incomplete programs, 
and to construct entire 'programs. 

A tw9-vay a^rialysii^ of variance was performed on the task, data, 
measuring the effects of treatment and sex. The. results are summarized 
in Table 2. Tfee experimental group 13 labeled "TSA" since their tasks 
were selected by BTP's task selection, .algorithms ; the "path" group 
followed the predetermiRed branching, strategy through^the curri-c^lum.- 
Some conclusions about these rccults can. be .drawn from the analysis ■ 
completed at this, time. First, thei-e was no significafnt difference 
between, the groups.* posttest scores (means were IO9..O and 108,.2 for the 
experimental and control groups, respect! vely) , indicating that the two 



TaBle 2 

Mean Performance qf Experimental ( taa) and Conti^ol (*path) 
Subjeqts Collapsed Across Sex .\ 
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. .,1. 
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• 


tsa 


path 


■ \ 




Total number of tasks seen 


37.7 


29.k ^ 


8.3 • 


. 21. 92 


^ passed checker 


. 99:1 


82'. 3 




6.32 * 


^ passed checker "the first time 


73.0 


62.3 . 


10.7 


-j.hl* 


io "understood" in PTI 


-91.1 


83.2 


,■ .7.9 \ 


-6.68 */. 


where model solution was seen , 


. 10.8 


Ik. 3 

• 




.85 



* ^crit ^^'^'^^ = ^-17, P <- .05 
(i;3S) = 5. '57, P < .61 



task selection strategies . apparently prpdueed .the same amount of .learning, 
of the material tested. However, the data in Table 2 shov a significant 
difference in. the character of that lemming experience. During their 
ten contact hours, i::tudeaits in the ^experimental group wcwrked 2^1^ mor^ 
problems than those vho followed the predete mined problem sequence, 
and had, significantly less trouble working the problems they were p3je- 
sented, as evide'nced' by the higher percentage -they completed correctly 
and said they understood in "the post task, interview. It should be 

* Stressed' that the two groups saw the same problems, but in a different 
'sequence.' Neither the evaluation forms filled out after each task nor 

, their. posttest scores indicated that they were getting problems that, 
were too easy. We believe that these results show^^that BIP|s task 
selection algorithm did indeed choose appropriate .problems for. each • 
student on the basis of ' a running history-, and that the net effect, 

' 'although it. fell short of increasing learning speed in this situation, 
was a significant -improvemeiit over the branching strategy devised by 

. our experts.' 

The data- coXlect ion routines were designed to 'be nearly exhaustive, 

* . * . ., . ■ ^ 

recording all information that we felt might be" interesting in some 
aspect of future work on the design of task selection strategies, student 
models, and curriculum description; for this' reason, we feel that, much • 
moi^e/is yet to ber4feriv-ed from .the data than the/ results we give here. 
StillV we are confident that ^ HP's strategy, 1^ ordering the presentation 
tasks jan-j:he basis. of its' continually updated knowledge of each 
\.,.^^'iect's.. progress, did chang^-Jihe^charap'ter of the Interaction between 
'the .teaching syistem, and -the-studfiSEs. ' \ 



It is ou'r goal to improve student performance significantly . by 
tailoring the presentation of • tasks more closely to each student's - 
strengths, and weaknesses/ and we "^f eel that the experimental results in- 
dicate a positive first step in that direction.- Future work will focus 
on the three major components of the individualization process: the • 
curri'eulumo description, the student model, and the task sjelection algo- . 
rithm o,r strategy. In order for BIP" to present a reasonable task, ^the 
curriculum must be represented internally such that BIP can recognizfe . 
the aspects -of each task that qualify it, as appropriate for the^ student »s 
current level of ability. The description pf the student, similarly, 
^must represent his abilities in- a fonuat that rirakes it possible for BIP 
to^'identify the kind of task -that is needed. Finally, the task selection/ 
strategy must serve as an interface between the two descriptions, defin- 
ing an ■api)ropriate task in tems of . the current state of the two data 
bases. Further analysis of these data is "planned, not only to discern 
other differences between the two groups. Mp're/^^importantly, we' will. use 
Lhe results in designing. new f orms- ot the three components of the in- 
dividualization scheme. ' ' . .... 

We are also concerned with the difficulty and cost of running these 
large soale comparative experiments. At least one staff member was 
absnr^ed hy the logistics of the February experiment ^during the recruit-^ 
ing and pretesting period, three weeks of classroom interaction with BIP, 
and the time/ required for administering, posttests. • Considerable effort 
was expended to insure that the routines that controlled the two treatment 
groups worked properly, and to insure .that the data were .collected 
exactly, as planned. - No changes could be made to these routines daring • 
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the subjects' work vlth JSCP, so it vias necessary (as i^a any live-subject 

e3q)eriment) to devote about a month to the perfection ofXthe program. 

.The information collected, vhile extensive and very useful{\for a number 

of purposes, still relates only to tvo conditions, only one ^f • vhi'ch 

(the "tsa'' group) is of interest for future development* ' \ . 

/. Still, such comparative studies should, be run ''for each-.of i^he many 

design decisions made during the development of the student model^^and 

■ ■ . * ' , ■ ■ . \ . 

task selection algorithms.^ An alternative means of generating experi- 

mental results is needed^ to provide efficient comparison .an(J > valuation ^ . 

of our designs.. We are "developing a procedure for obtaining^ detailed / 

information about KEP'^ ability tb- individualize instructioh by simulati^ng . 



\ 

\ 



large-scale experiments like this one instead of actually carrying them \ 
out. With the simulation we expect to obtain reasonable data abotit ne^ \ 
student models as they evolve • 39.- that future real-subject experiments 
focus on more specific evaluation^ of the task selection process* 
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. ' • APPENDIX A 

BIF's ' Curriculum Structure • 

• • ■ " , ' ^ '■ * " . . 

It ^is. a difficult task to detemine thq fundamental elements of ai^- 
arbitrary curriculum' and to discern or impose a structure. The ^ail 
of description required by BIP's task selection algorithms seemed im-_ 
•possible to attai^ and indeed we ha 'never slttled. down tV 
'.satisfactory description of. the introductory programming cumculum, 
although we do feel that- we understand it better; than we 'did. ' We have 

included in this appendix a list of^ost of the* basic skills we have . 

'-^ V ' / • 

identified, grouped by the. subgoals or "techniques" they first appear 

^ \ ' ' ' ' ■ ' ' • * . 

in* Some' ^^ski lis, like "the .END statement"^ are not/ included in the - 

* " , • ' ' ' ' ■ ■■ . ■ 

.technique structure because they are not valuable . in tasH selection 

:decisions/ Although this description is neither complete, nor final, 

it cert^tnly represents cons,ilerable "effort and should be veiy valuable 

to others; interested in teaching computer programming' or in curriculum 

analysis, ^ . ' • 

Simple butput - first programs ^ . ' ; ^ . , 

* ^ . ^' , • . ' ' " ■ V . ' " ^ ' ■ ' ' ' ■ 

print numeric literal ^ . . • 

print string, literal ' , 

print n^eric expression [operation on literals] - ' -% 

print stilng expression [concatenation, of literals] ^ 

Variables assignment- ' . \ . " . * , 

\ ' * ' i ■ * ' 

print value of numeric variable^. ^ ' . 

' print value of string' variable * ■ " . ' - 

print -numeric expres'^ioh [operattion variables]* 
print numeric expresision [operation on literals and. valuables] 
print string expression [tsbhcatenation" of . variables] . / " 

print string ^expression' [concatenation; of variable and literal]. 
*assign value to a numeric variable, [literal value] 
•assign value to a string variable; [liiberal value] ' ; . • . 



felore complicated assignment 



assign to a string variable [value of an expression] 
assign to a numeric variable J value of • an e.xpression] / 
re'-assignment of variable (using its own value) [siring] 
re-assignment of variable (using its. own value) [numeric] 
assign to numeric variable the- Value of ahother variable 
assign to string variable the value .of another variable 



More complicat.4d output . ; 

multiple print [string literal, numeric variable] 

multiple print . [string literal, numeric variable expi^fession] 

multiple print [string literal^ string variable] 

multiple print [string literal-, string variable exp^ssion] 

Interactive programs - "^INPUT from User - usi^g DATA^ 



assign numeric valuable by -INPUT- ^ , ' 

assign string varim)ie by -JNPUT- ;\ 
as'sign numeric variable by -READ- and--DATA- 
assign string variable by -READ- and -DATA- 
the -REM- stateme.n;t 



More complicated* Input 



mu*ltiple 
y multijy] 
/ mult^r^e 
mirltiple 
* multiple 
^multi^ple 
multiple 
multiple 
multiple 



Lues ''in • 
values in - 
values rn < 
assignment 
Assignment 
assignment 
as3ignment 
assignment 
assignment 



•DATA- [all numeric] 

■DATA- [all string] ^ ; 

•DATA- [mixed numeriq and sTbring] 
by -INPUT- [numeric variables] ' « - ^ 
by -ITfPUT- [string variables] 
by -INPUT7 [mixed numerj,c/ and string] 
by -READ- [numeric "variables] 
by -READ- [string' vailabies] 
by -vREAD-: [mixed numeric and string] 



Branahing r program flow 

' unconditional branch (-GOTO-) 
•interirupt with ctrl-g ' 



Boolean' expressions . > 

print boolean expression [relation of string literals] 
. print boolean expres-sion [relation of numbric literals] 
print boolean expression ^ [relation of numeric literal and variable] 
print boolean expression /[relation of stating* literal and variable] 
boolean operators [-AND-] 
boolean operators J-OR-] / ' 

boolean operators [-NOT-] 



V .. 



IF statements - conditipnal branches * ... 

. . V • ' 

conditional irancho [compare numeric variable with numeric literal] 
conditiotial' branch [compare numeric variable with expression] , ^ 
conditional branch [compare two numeric variables] . ^ 
conditional branch' [compare string variable with string literal] 
conditional branch [compare two string variables] 
.the -STOP- statement ' . , . •. 

'Hand-made loops - iteration ' * ^ ^ . 

conditional branch [*compa're countejp with numeric literal] . 

; conditional branch [compare counter with numeric variable]. 

initialize"" counter variable with a literal V^liie 

initialize. counter variable with the value ot a variable 

increment (add to) the' value of a counter variable 

decrenint. (subtract from)' the value of a, counter variable. 

.• \^ . ■ • 

' ' . ' ' ^ - - ■ • ■■ " 

Using loops to- accumulate / . 

accumulate successive values into* numeric variable 
accumulate • successive values into string va-riable 
calculating'bomplex' expressions [nurAeric literal and variable] 
initialize numeric* variable ('^not counter) to literal value 
initialize numeric variable (not counter) to value •of a variable 
initialize string variable to literal value v 
initialize string \rariable to value of a variable* ' 



J 
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BASIC functionals 

' . the -INT- function* ' / .: ■ 

the -RND- function 

' the -SQR- function. , . 

FOR...MIXT loops ' • ' ■ 

, , *■ ■ • ■ 

FOR . . WXT loops with literal- as final valjie of iTidex 
FOR . . NEffl^ loops with variable as final- value of index 
FOR . .,KE}CT loops with positive step size other than 1 
' . FOR . . NEXT .loops with negative step size . 

• Arrays . • . * 

assign element of string array variable by -INPUT- ' 
■ assign element of numeric array variable by -INPUT- . ' , 

assign element of numeric array variable [v^lue is also a variable] 
the -DIM- statement ■ ' ,. , 

. string array using numeric variable as index 

.print value of^ an element of a string array variable 

numeric array using numeric variable as index 

print value of . an element 'of a numeric array variable 



Puturfe extensions to the curriculiim 



nesting loops (one loop inside another) ^ 
subroutines ( -GOSUB-' and friends) 
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